




T 

t- 



l 



T ~ ■ 



,.^1. iJ 



DOCUMENT RESUME 

ED 025 785 cc 003 012 

ByLeooiog, Oscar; And Others 

Student Educational Growth During the First Two Years of College. 

American Educational Research Association* Washington* D.C. 

Pub Date 7 Feb 68 

Note" I8p.; Paper presented annual meeting of the American Educational Research Association* Chicago* 
Illinois* February 7* 196^ 

EDRS Price MF -$0.25 HC -$ 1 .00 

Descriptors" Academic Achievement* ♦Achievement Rating* ♦Achievement Tests* ♦College Students* 
♦ Comparative Analysis* ♦Student Development 

Identifiers- American College Tests* School and College Ability Tests* Sequential Tests of Educational Progress 

This study is concerned with educational growth during the first two years of 
college* as measured by the American College Testing Program. Students from five 
colleges were pretested during the senior year of high school and post- tested after 
the sophomore year of college. Factor analysis and t- tests showed general gains in 
the four subtests: mathematics* English* natural sciences* and social studies. Sex 
differences in gain patterns occurred. Institutional differences influenced growth in 
specific subjects. The educational growth of a student was unrelated to his major. 
Both initially high and low scoring students regressed the the mean on the re-test. A 
study of the students in one college on the following aspects is also reported: Cl) the 
effect of taking a specific college course* (2) the influence of a student’s major* (3) 
the gain at the end of the freshman year* and (4) the results of the Sequential Tests 
of Educational Progress and the School and College Ability Tests. The total study is 
considered preliminary due to problems such as regression* ceiling* content* and 
reliability effects. (NG) 



mmm 



I 




i 






















Student Educational Growth During the First Two Years of College^ 

Oscar T* Lenning, Leo A. Monday 
American College Testing Program 
and E. James Maxey 
Measurement Research Center 



The purpose of education is the growth of students* Students 
should change for the better, or "grow," as a result of their educational 
experience* But how, exactly, should students change? Few people 
would quarrel with the notion that, among other objectives, students 
should grow "educationally"; i*e*, as a result of college experience, 
students should demonstrate a greater knowledge of subject matter, more 
skill in use of language, and increased reading ability— to read with 
comprehension, to apply their readings to new situations, and to recog* 
nise writers' styles and biases* Further, they should be able to ana* 
lyze and solve problems, to make inferences, and to think critically* 
These are abilities measured by the ACT tests. 

Using ACT Tests to Measure Growth 
The ACT tests, we therefore believe, may be useful for measuring 
the educational growth of college students* Designed for college 
admissions, the ACT tests are usually administered to college*bound 
students in the senior year of high school* The tests are generalized 
measures of educational development in the subject matter areas of 
English, mathematics, social studies, and natural sciences* As such 



^Paper prepared for the symposium College Student Growth and Devel* 
opment presented at the 1968 annual meeting of the American Educational 
Research Association, February 7, 1968, Chicago, Illinois* 
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the tests are more curriculum-oriented than scholastic aptitude tests 
that yield two highly generalized scores, one verbal and one mathe- 
matics, but less curriculum-oriented than some tests available at the 
secondary school level. This is necessary because a college admissions 
test cannot be so curriculum- oriented that It gives a systematic advan- 
tage to students using a specific textbook or who are In a particular 
curriculum. The more curriculum-oriented a test, the more we would 
expect It to be sensitive to instruction and to specific student 
growth. Because the ACT tests are somewhat curriculum-oriented and are 

1 

at least moderately correlated with academic success In college, we 
could logically try measuring change with such Instruments. Conser 
quently, our concern here Is with the measurement of educational 

- 

growth during the first two years of college and the usefulness of the 
ACT tests for this kind of assessment. 

I 

Problems In Measuring Student's Growth 

An obvious way to measure change would be to administer retests 

after the first and/or second year of college to see If there Is any 

gain. But few areas of educational research have been troubled with as 

2 ^ 

many statistical design problems as the analysis of student change. 

Ve can do little more here than list some of these difficulties to 

I 

alert us to potential problems. These difficulties are regression 
effect, celling effect, content effect, and reliability effect. 

Regression effect refers to the fact that on retest students tend 

i 

to go back or regress to the. mean for the student sample. A student 

j 



In a useful book on this subject, Harris (1963) edited a series 
of papers on the measurement of change. Researchers Interested In this 
area are encouraged to study this book. 
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with a high initial score will ter«u to make a lover score (regress 
to the mean) on retest. Similarly, a student with a low initial score 
will tend to make a higher score (regress to the mean) on retest. If 
we desire, then, to compare mean change for two groups of students 
(e.g., those who had a math course and those who did not), it would be 
appropriate to adjust the retest scores across the two groups by the 
initial scores. Otherwise, since the more able students would likely 
be taking math and the less able would not, regression effect might 
lead us to conclude the less able ones with no math course made the 
bigger gain when, in fact, there was no true difference in gain 
between the %vo groups. 

Ceiling effect means ^hat a student with a high score on a test 
has little room to go up, in comparison to a student with a low score. 

Content effect refers not to general statistical problems but to 
the content of college admissions tests being more closely parallel to 
that of college courses for average and below average students. College 
admissions tests are built to be suitable for a wide range of talent, 
with item difficulties and test content appropriate for college-bound 
high school seniors. Such tests may not adequately cover college 
course work in honors programs. For example, a college may section 
in English; and the ACT English test, with its emphasis on grammftr 
and detecting bad writing, would be appropriate for average and below 
average sections. But the upper sections may study literature. Lit- 
erature is not covered in the ACT tests. On retest the students who 
were initially low and average would be expected to show gain because 
they had Just been studying this material. In contrast, the initially 
high students would show little or no gain and perhaps even regress. 
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Since they had not been studying the material in college, the test 
for them would be more one of memory than of English grammar and writing* 
Following this rationale for the four subtests, more gain would be 
expected during the freshman than the sophomore year of college, be» * 
cause the test content more closely resembles college freshman courses 
than the more diverse and advanced sophomore coursework* 

Content effect as well as ceiling effect and regression effect 



favor students witli below average initial scores; these students would 

1 

show the most gain on retest* Many initially high scoring students 

: 

will make lower scores on retest. j 

Reliability effect means that two d«.sirable characteristics of 
college admissions tests produce difference scores between two testings 
of low reliability* These two characteristics are (1) highly reliable 
scores and (2) highly correlated test-retest scores obtained within a 

i 

year or two of each other* In short, the psychometric requirements for 
a good college admissions test are different from those for a good test 
to measure change* And the ACT tests are constructed to be good college 
admissions tests* 



We could react in two extreme ways to these difficulties* First, 
we could decide that the statistical, psychometric, and educational 
problems In measuring student growth are so great that they preclude 
educational research in this area* S'^veral researchers have taken this 
approach and rejected this area for study* Second, we could say the 
problems are great and unsolvable, so we might as well go ahead and 
ignore them in our research since we cannot do anything about them 
anyway* We resolved to do neither, not giving up and not ignoring the 
inherent problems • 
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Questions Colleges Have about Student Growth 

As a part of their institutional self-study research programs, 
many colleges and universities have preferred objective measures of 
student educational development. The following questions typify those 
asked by college and university administrators: 

1. As a result of college experience do students, on the average, 
learn something, and can they better apply and make use of 
what they learn? In operational terms, did they make higher 
scores on retest? 

2. Is the pattern of educational growth different for. men and 
women? 

3. If a student has completed a course in a certain subject 
matter area, does he do better on retest in this area than 
if he had not taken such a course? 

4. If a student majors in a certain subject matter area, does he 

do better on retest in this area than if he does not? 

5. Is the average growth during the college sophomore year of 

the same magnitude as during the college freshman year? 

6. How does student growth at our college compare with that at 
other colleges? 

7. How much gain is expected at different initial score levels? 

8. Do other related tests measure educational change in college 
students? 

Design 

This research was conducted at five institutions located in Ark- 
ansas, Iowa, South Dakota, Texas^ and 'Wisconsin. Two institutions were 
state colleges, one was a private four-year college, one a Junior college. 
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and one a state university. The ACT pretest was written for college 

admissions purposes, typically during the senior year in high school. 

The retesting* using a different but parallel ACT tettform, occurred after 

the students had completed two years of college. One college also tested 

students at the end of one year of college. And one college also had 

available STEP scores (level 1) on test-retest, while another had SCAT 

scores (level 1).^ Institutions indicated student sex, whether or not 

students had had a relevant course in each subject area of the tests, 

4 

and the students' educational majors* 

To determine whether students on the whole made gains, t, tests were 
conducted on the gain scores, separately for the male and female samples, 
for each college. Two-factor analysis of covariance, with one factor 
being sex, and the other being the four ACT posttest scores adjusted 
by respective initial scores across sex, was used at each college to 
determine if there were differential gains for men and women (Winer, 

1962, pp. 602-618). Two -factor analysis of covariance was also used 
to determine if gain was different for the college sophomore year than 
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the college freshman year* One factor was year, the other was ACT- 
posttest scores adjusted across year for initial scores* Covariance 
was used to adjust for differences between groups (men and women in 
the first case, freshmen and sophomores in the second) on initial scores 
so that possible differences between groups on initial scores would not 
confound the results. One-factor analysis of covariance on posttest 

^The STEP (Sequential Tests of Educational Progress) and the SCAT 
(School and College Ability Tests) are published by Educational Testing 
Service, 

^The information available varied with institutions* The only data 
common to all five colleges were sex and ACT test-retest scores. 
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scores (Cooley and Lohnes, 1962, pp. 63«86) was conducted for each ACT 
subtest to determine if there were differences between (a) students who 
had a relevant course or not, and (b) students who were majoring in the 
relevant curricular area or not. If there were more than two groups 
being compared, follow*up ^ tests were used to compare adjusted group 
means (Snedecor and Cochran, 1967, pp. 429-430, 441) whenever a signi- 
ficant F was obtained. Another procedure involved preparing a table to 
show average observed gain (or loss) on posttest at the end of two 
years of college for each initial standard score on each ACT test. And 
finally, t tests for the available STEP^ and SCAT scores means, pre and 
post, were conducted to determine the educational change measured by 
these instruments over two years of college. STEP and SCAT tests wers 
first administered to college freshmen soon after they arrived on cam- 
pus and then to the same students after two years of college. 

Results 



To investigate observed gains on the ACT tests, ^ tests were con- 
ducted for each of the five campuses. Results are shown in Table 1. 



insert Table 1 about here 



There were significant gains (P5.05) in almost every case for ACT Social 
Studies, Natural Sciences, and Composite scores. In general, ACT test- 
retest mean differences were found on most ACT tests at most institutions 



with especially large gains being noted for r-oclal Studies and Natural 
Sciences. It is also apparent that there are institutional differences; 
e.g,, the college with the most highly significant mean gain surprisingly 
had the highest initial means. 



^Data were available for only five of the STEP tests: Reading, 

Writing, Mathematics, Social Science, and Science. 
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Table 2 lists the average observed change for each subtest at each 

Insert Table 2 about here 

initial standard score for the entire group of 972 students who were 
retested at the end of the sophomore year. As expected, those with high 
initial test scores had low and negative change on retest, while those 
with low initial scores had large positive score change. Table 2 demon- 
strates the need to control for regression effect when between-group 



comparisons are made. 

The remaining results of the study will be summarized briefly and, 
for simplicity, without reference to tables. A two-factor analysis of 
covariance was computed at each institution to determine if there were 
different gain patterns on the four tests for men and women. Post test 
scores were adjusted on initial scores in order to avoid giving either 
men or women a systematic advantage due to regression. At four of the 
five institutions interaction was found, suggesting that there were 



different gain patterns for the two sexes. Follow-up tests using one 
factor analysis of covariance indicated a sex difference only on the 



ACT Mathematics scores at three of the institutions and on all suotest 
scores other than Mathematics at the fourth college. In these cases, 
where significant difference occurred, males had more gain on ACT 
Mathematics, Social Studies, and Natural Sciences, and women had more 



gain on ACT English. 

Next, students who had taken a relevant course were compared on 
the appropriate ACT test with students who had not taken such a course 



®A supplement containing all tables for this study is available 
to Interested persons. Address inquires to ACT Research and Develop- 
ment. Box 168, Iowa City, Iowa 52240. 











Page 9 



or who had taken fewer such courses* Analysis of covariance was used, 
with posttest scores adjusted on pretest scores* On every campus, 
almost all students took Engliish and the aame amount of English course- 
work during the first two years of college* Consequently, we could not 
study the effects of an English course on ACT English mean gain* 

Three colleges reported data to permit a study of the effect of 
taking a mathematics course on Mathematics test scores* On all three 
campuses, students who took a mathematics course made greater gain than 
those who did not* At two of these colleges, students were divided in- 
to three groups-those with no courses, those with one mathematics course, 
and those with two or more mathematics courses* Consistently, the 
more mathematics coursework, the greater the mean gain on the ACT 
Mathematics test* 

Few students had not taken a social studies course* Data for only 
one college permitted sectioning the students into a "no social studies" 
group- and a "some social studies" group* At another college, data 
permitted a breakdown into "0-3 semester hours of social studies" and 
"6 or more semester hours of social studies*" Mo significant mean 
difference on ACT Social Studies scores was found at either campus* 

Three colleges reported data for natural sciences* One college 
reported whether or not students had had a science course* Students 
who had a course made a significant gain on t^test* Another college 
found almost all students had had a science course, so students were 
broken into two groups-those who had from 0 to 5 semester hours of 
science, and those who had 6 or more hours* There waa no significant 
difference on retest for the two groups* At the third college, groups 
Included students with no science courses, those with one science course^ 
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and those with two or more science courses. There were no significant 

differences in gain for the three groups. 

The effect of educational major on test scores was examined next. 
Three colleges reported the majors of their students. Taking each of 
the four ACT tests one at a time, we investigated whether gain on a 
test was related to educational major. On the Mathematics test on one 
campus, mathematics and science majors made significantly greater gains 
than English and language majors, social studies majors, education 
majors, and business majors. On the other campus, secondary education, 
business, and liberal arts majors made significantly greater gains on 
the llathematics test than elementary education majors. At one college, 
gain on the Natural Sciences test scores was related to major. Math- 
ematics plus science majors and fine arts plus no-majors made signifi- 
cantly greater gain on the Natural Sciences test than education majors. 

Next we compared growth, separately for males and females, during 
the college sophomore year with that during the freshman year. Data 
were available from only one college where there had been a retest 
of one group at the end of the freshman year and of another group at 
the end of the sophomore year. A two-factor analysis of covariance 
with a one-factor analysis of covariance follow-up (separately by 
ACT subtest) was used to determine if the growth pattern was the same 
for the two groups. The within factor was "ACT subtest" and the 
between factor was "sex." No significant interaction was found for 
females, and it was just barely significant at the .05 level for males. 
Follow-up tests on males revealed no significant differences on specific 
subtests. Therefore a "year effect" was present for both sexes; most 
of the gain occurred prior to the sophomore year of college. 
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One college included SCAT scores, another STEP scores on test- 
retest. Significant gains were found for the SCAT Verbal and Total 
scores. For the same sample, significant gains were found for the ACT 
Social Studies, Natural Sciences, and Composite scores. The college •. 
that studied STEP found significant gains on all STEP tests studied: 
the STEP Reading, Writing, Mathematics, Social Science, and Science 
scores. This same college found signf leant gains on the ACT Mathe- 
matics, Social Studies, Natural Sciences, and Composite scores, but 
not on the English scores. 

Discussion 

At this point let us briefly summarize our findings. 

1. In general, students tested at the end of the college sopho- > 
more year show gains on the four ACT tests. ACT English scores 
seem to be the least sensitive to student growth while Social 
Studies, Natural Sciences, and Composite scores are the most 
sensitive. On an institutional level, English and Mathematics 
are the only two that have any negative mean score changes. 
English and Mathematics are probably more influenced than the 
othv^r two by content, ceiling, and regression effects. 

2. Institutional differences apparently determine growth and the 
subject matter areas in which it occurs. For example, the 
college with the highest initial score means had the greatest 
observed gain by far on ACT Social Studies and Natural Sciences 
scores. This difference in gain would be even more marked if 
there were no ceiling and regression effects present. In 
contrast, this college had less English score gain than did 
the others, and for females the change was negative. Such 
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Institutional differences may be the result of student Input 
characteristics or of campus atmosphere and Instructional 
characteristics y or of both. 

3. Different gain patterns on the tests are found for boys and 
girls. At three of the five colleges • boys gained signifi- 
cantly more on >Iathematlcs scores than did girls » possibly 
because more boys take mathematics courses than girls. A 
fourth college had significant sex differences on the other 
three subtests, excluding Mathematics. 

4. Students who take a college mathematics course show gain on 
ACT Mathematics test scores. No such growth Is detectable 
for students who take a social studies course. ACT Social 
Studies scores may reflect such content as generalized reading 
comprehension which Is less related to social studies courses 
than, for example. Mathematics scores are to mathematics 
coursework. Seml“tebhnlcal reading and comprehension skills 
are emphasized and practiced In the freshman courses of many 
curricular areas. Social Studies scores have proved to be 
predictive for a wide range of freshman courses. Thus, we \ 
might expect much overall gain on this test for the college 
population as a whole with little difference generally 
apparent when mean change comparisons between groups of 
students are made. The available data did not permit an 
adequate study of the effect of having a college English or 
science course, but there would likely be more gain on ACT 
Natural Sciences scores associated with taking a science course 
than gain on ACT English scores associated with taking an Eng- 



lish course. 
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5. Generally, student 's educational growth Is unrelated to his 
major. Gain on Mathematics test scores do seem to be related 
to major in some instances, and gain on Natural Sciences tests 
may be. Little change as a result of major could be due to 
several factors. Majors have already mastered most of the 
basic skills covered in that test and thus have much less room 
for improvement (ceiling effect). And most students may not 
have decided on a major until shortly before retest and thus 
have had no additional coursework in their major field. It 



is also common practice to leave most courses in students ' 
major areas until the Junior and senior years of college. 

6. Students tested after the sophomore year did not show signi- 
ficantly more gain on any of the subtests than students tested 
after the freshman year. It has been mentioned that the ACT 
tests were built to predict first term grades and are more 
applicable to first year courses. Perhaps the more advanced 



courses of the sophomore year do not emphasize much of the 
test content. Sophomore students therefore do not have the 
opportunity to increase proficiency in such skills. During 
the freshman year most students may near their peak proficiency 
on the basic skills emphasized in the tests. 

7. Negative gain (or loss) is associated with initially high 

scoring students while positive gain is associated with initially 
low . scoring students, as expected because of regression effect. 
Mean gains occurred on Social Studies and Natural Sciences test 



scores at a much higher initial score level than for English 
and Mathematics scores, implying the presence of content effect. 












mmmmm 






1 



Page 14 



r. 




8* Gain In scores over two years In college Is detected on both 
the SCAT and STEP tests* 

In addition to the problems in measuring change, there are certain 
other limitations to this study* First, we used gain on the respective 
subtests to study the effects of courses and majors, when it might have 
been better to construct variables of "potential" in the four academic 
areas of the tests, based on the optimal weighting of the tests used 
routinely for the institutional prediction of college grades* In other 
words, we used content rather than predictive validity in dealing with | 

educational development in the four areas of the tests* Second, there 
was no control for maturational effects* ACT-tested students who did j 

not go to college could be re-tested at the end of the same time inter- 
val and their changes compared with college students* Third, there 
was no control for student input on an inatutitional level this was 
Justified on the theory that institutional differences were not of 
interest in this study since we were focusing on the educational develop- 
ment of college students in general* Fourth, almost a year of high 
school experience occurred after the pretests* A study with the pre- 
test given at the beginning of the college freshman year would avoid 
this problem* Fifth, the motivation to do well on the tests may have 
been considerably greater when the tests (pretest) were written for 
college admissions, than Vhen the post test occurred* 

However, this study of the educational growth of college students 
was intended to be a preliminary one* We consider the ACT tests along 



Stanley (1966) has discussed the problem of controlling for student 
input in college effects design* For an example of one experimental 
approach, see Nichols (1964)* 
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with the SCAT and STEP as being of some use in this case. 

Additional research in college student growth could daul with in* 
stitutional differences in average student educational growth, student 
differences in educational growth, interactions of students and institu* 
tiona on growth, and special applications of our knowledge to assessing 
growth among culturally and educationally disadvantaged college students* 
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TABLE 2 






DISTP'^BUTIOI'I ACCORDING TO STANDARD SCOPJ2 ON THE PRETEST 



OF ACT TEST-RETEST KEAN CHANGE^- 



Statistics for the entire group of students (972 students from 5 colleges): 

m 

Natural 
Science 

19.98 

1.92 

.67 



Pretest mean 



Mean change 



12 



English 

19.11 

0.A6 

.70 



Mathematics 

18.75 

0.82 

.77 



Social 

Studies 

19.96 

2.A8 

.71 



pretest 
Standard 
be ore ^ 


ACT 
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Math 


Freq 
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ACT 


Compos 
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Sc£i:TI 


Freq_ 


Mean 

Chan?,e 


Freq 


Mean ‘ 
Change 


Freq 


Moan 

Change 


Mean 

Freq Change 
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36 
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4 


-2.00 
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|34 






3 


-2.33 














34 


33 






4 


-4.00 


1 


-2.00 


5 


-2.40 






33 


32 






6 


-3.33 


8 


-0.50 


3 


-1.00 






32 


31 






11 


-1.18 


10 


-1.80 


4 


-1.50 


2 


-1.00 


31 


30 


1 


-5.00 


7 


-2.14 


17 


-1.59 


14 


-1.29 


4 


-0.50 


30 


29 


5 


-2.20 


19 


-1.84 


16 


0.06 


18 


-1.39 


6 


-0.50 


29 


28 


11 


-1.64 


22 


-1.73 


24 


-0.75 


37 


-0.89 


20 


-0.35 


28 


27 


20 


-1.85 


36 


-0.83 


26 


-0.54 


39 


0.10 


26 


-0.04 


27 


26 


36 


-1.72 


38 


-0.18 


43 


0.58 


61 


-0.49 


29 


-0.07 


26 


25 


40 


-2.05 


42 


-0.36 


74 


1.68 


34 


0.21 


37 


0.27 


25 


24 


57 


-1.16 


35 


-0.94 


58 


1.90 


69 


0.29 


65 


0.52 


24 


23 


68 


-0.66 


62 


-0.32 


70 


1.33 


71 


1.17 


56 


0.95 


23 


22 


67 


-0.66 


31 


0.55 


49 


1.35 


49 


0.86 


85 


1.07 


22 


21 


79 


0.11 


39 


0.15 


64 


2.03 


74 


0.76 


91 


0.89 


21 


20 


88 


-0.11 


50 


-0.36 


80 


2.26 


66 


1.30 


87 


1,06 


20 


19 


92 


0.04 


58 


0.17 


53 


3.43 


55 


1.62 


79 


1.48 


19 


18 


93 


1.32 


77 


0.29 


58 


2.34 


71 


2.65 


92 


1.84 


18 


17 


74 


1.39 


59 


1.08 


55 


4.05 


42 


3.55 


73 


2.19 


17 


16 


58 


0.76 


67 


1.42 


59 


3.24 


42 


3.12 


46 


1.91 


16 


15 


44 


1.30 


67 


1.43 


41 


4.90 


43 


4.72 


49 


2.14 


15 


14 


32 


2.00 


46 


1.26 


46 


4.76 


45 


4.02 


34 


2.26 


14 


13 


28 


2.93 


46 


1.80 


39 


4.00 


28 


5.14 


26 


3.54 


13 


12 


15 


2.20 


40 


3.12 


14 


4.36 


32 


5.03 


25 
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11 


10 
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3.50 
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4.11 


14 


4.36 


9 
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3.93 
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3 


2.67 


8 


7 


1 


7.00 


17 


4.47 


7 


9.29 


5 


8.40 


2 


4.50 


7 


6 


7 


7.00 


7 


4.86 


6 


7.17 


7 


9.29 


2 


8.00 


6 


5 


1 


6.00 


7 


9.86 


1 


4.00 


3 


10.33 






5 


4 


1 


13.00 


1 
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3 


2 


2 


9.50 


2 


7.00 
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16.00 


1 


6.00 






2 


1 


1 
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1 



Composite 

19.58 

•1.42 

.84 



^All students in the sample had attended tvo years of college prior to the posttest. 



bACT tests have a standard score range of 1-36. with the scale being the same for the fouil 
tests and composite. National college bound norms for the ACT tests show means and 
standard deviations of approximately 20 and 5 respectively. 













